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Abstract 

Background: Healthcare claims databases have been used in several studies to characterize the risk and burden of 
chemotherapy-induced febrile neutropenia (FN) and effectiveness of colony-stimulating factors against FN. The 
accuracy of methods previously used to identify FN in such databases has not been formally evaluated. 

Methods: Data comprised linked electronic medical records from Geisinger Health System and healthcare claims 
data from Geisinger Health Plan. Subjects were classified into subgroups based on whether or not they were 
hospitalized for FN per the presumptive "gold standard" (ANC <1.0x10 9 /L, and body temperature >38.3°C or receipt 
of antibiotics) and claims-based definition (diagnosis codes for neutropenia, fever, and/or infection). Accuracy was 
evaluated principally based on positive predictive value (PPV) and sensitivity. 

Results: Among 357 study subjects, 82 (23%) met the gold standard for hospitalized FN. For the claims-based 
definition including diagnosis codes for neutropenia plus fever in any position (n=28), PPV was 100% and sensitivity 
was 34% (95% CI: 24-45). For the definition including neutropenia in the primary position (n=54), PPV was 87% 
(78-95) and sensitivity was 57% (46-68). For the definition including neutropenia in any position (n=71), PPV was 
77% (68-87) and sensitivity was 67% (56-77). 

Conclusions: Patients hospitalized for chemotherapy-induced FN can be identified in healthcare claims 
databases-with an acceptable level of mis-classification-using diagnosis codes for neutropenia, or neutropenia plus 
fever. 
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Background 

Neutropenia is a common side effect of myelosuppressive 
chemotherapy. Neutropenia both increases the risk of 
infection and diminishes patients' ability to fight infection. 
When neutropenic patients develop fever, a cardinal sign 
of infection, the high likelihood of serious consequences 
usually results in hospitalization for urgent evaluation, 
ongoing monitoring, and administration of intravenous 
antibiotics. This common and well-studied complication 
of neutropenia is called febrile neutropenia (FN), regard- 
less of whether infection is ultimately documented as 
the cause of fever. FN— as well as severe or prolonged 
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dose-delays, dose-reductions, and discontinuation, inter- 
fering with the delivery of optimal treatment and possibly 
adversely affecting patient outcomes [1-4]. For these rea- 
sons, prophylactic administration of a colony-stimulating 
factor (CSF) -which has been shown to reduce the risk of 
FN, FN-related hospitalization, and infection, and to reduce 
antibiotic use in clinical trials— is recommended concur- 
rently with myelosuppressive chemotherapy when FN risk 
is estimated to be approximately 20% or greater [3,5-8]. 

Using public and private healthcare claims databases as 
well as hospital discharge records, a number of retrospec- 
tive studies have been undertaken to assess the clinical 
risk and economic burden of FN requiring hospitalization 
in clinical practice and to evaluate the comparative effec- 
tiveness of CSF agents against chemotherapy-induced 
hospitalized FN [7,9-14]. Because the study databases did 
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not include information on absolute neutrophil count 
(ANC) and body temperature, and because there is no 
specific diagnosis code for FN, various combinations of 
codes for neutropenia, fever, and/or infection were utilized 
to identify FN in these studies. The accuracy of these defi- 
nitions is unknown, however, and thus it is uncertain 
whether the methods of case ascertainment may have 
biased study results. 

To the best of our knowledge, only one study has 
evaluated the test characteristics of claims-based defini- 
tions for FN, and this study included a small sample of 
older adults with a single tumor type (i.e., non-Hodgkin's 
lymphoma) and evaluated only a single definition [15]. In 
light of the dearth of data on the accuracy of the claims- 
based definitions that have been used to identify FN 
hospitalizations, and because such data may be important 
in the design of future studies as well as in reducing bias 
in estimators based on definitions that are less than per- 
fectly sensitive and specific (i.e., <100%), a new evaluation 
was undertaken [16]. 

Methods 

Data source 

Data were obtained from the MedMining Database, and 
spanned the period January 2004 through April 2010. 
The MedMining Database comprises electronic medical 
records (EMR) for services provided by the Geisinger 
Health System (GHS) to more than 2 million persons as 
well as healthcare claims for services received within 
GHS (as well as selected other health systems) to the 
approximately 225,000 members of the Geisinger Health 
Plan (GHP). 

Data from the GHS EMR include: patient demographics 
(e.g., age, sex, race); ambulatory care visits and inpatient 
admissions (along with associated diagnoses [ICD-9-CM] 
and procedures [ICD-9-CM, HCPCS], and dates of ser- 
vice); medication orders/lists; and clinical laboratory and 
exam results (including white blood cell count, neutrophil 
count, body temperature, and dates of observation). Data 
from the GHP healthcare claims database include in- 
patient and outpatient diagnoses and procedures, out- 
patient drug utilization, and dates of service. EMR and 
healthcare claims data can be linked for all GHP members 
using unique patient identifiers, and all such data can be 
arrayed chronologically to provide a detailed longitudinal 
profile of medical and pharmacy services used within 
GHS by each GHP member. 

Patient-identifying information was encrypted or removed 
from the study database prior to its release to the study inves- 
tigators, as set forth in the corresponding Data Use Agree- 
ment. The study database has been evaluated and certified by 
an independent third party to be in compliance with the 
Health Insurance Portability and Accountability Act of 1996 
(HIPAA) statistical de-identification standards and to satisfy 



the conditions set forth in Sections 164.514 (a)-(b)lii of the 
HIPAA Privacy Rule regarding the determination and docu- 
mentation of statistically de-identified data (45 CFR 46 
§46.101). Use of the study database for health services re- 
search is therefore fully compliant with the HIPAA Privacy 
Rule and federal guidance on Public Welfare and the Protec- 
tion of Human Subjects. Permission to use the data for this 
study was requested by study investigators and granted by 
MedMining. 

Source population 

The source population included all GHP members, aged 
>18 years, who began one or more courses of myelosup- 
pressive cancer chemotherapy between January 1, 2004 
and April 30, 2010. Receipt of chemotherapy was ascer- 
tained based on the presence of one or more claims for a 
chemotherapy drug or administration, which were identi- 
fied using HCPCS and ICD-9-CM codes in the healthcare 
claims database. Chemotherapy agents were classified by 
level of myelosuppression— any vs none— based on expert 
opinion. Regimens including one or more myelotoxic 
agents were considered to be myelosuppressive. Evidence 
of initiation of a new course of chemotherapy was based 
on the earliest claim for chemotherapy during the study 
period that was preceded by a 60-day or longer period 
without any other claims for chemotherapy; the date of 
the earliest such claim was designated as the "index date". 
Among these identified patients, those with evidence of 
cancer as indicated by two or more medical claims with a 
qualifying 3-digit ICD-9-CM diagnosis code during the 
period beginning 60 days prior to the index date were 
selected for inclusion in the source population. For 
patients with multiple courses of chemotherapy (based on 
a gap of >60 days between the last administration of 
chemotherapy in a given course and the first administra- 
tion in the next course) during the study period, all 
courses were considered. We excluded patients from the 
source population if they were not continuously eligible 
for comprehensive medical and drug benefits throughout 
the 60-day period prior to chemotherapy initiation. The 
classification of chemotherapy agents by level of myelo- 
suppression is available in Additional file 1: Table SI of 
the online supplement; cancer codes are available in 
Additional file 2: Table S2 of the online supplement. 

Chemotherapy cycles and regimens 

For each patient in the source population, we identified 
each unique cycle within each course of chemotherapy. 
The first chemotherapy cycle was defined as beginning 
with the date of initiation of chemotherapy (i.e., the index 
date) and ending with the first service date for the next 
administration of chemotherapy (as evidenced by a med- 
ical claim with a corresponding HCPCS or ICD-9-CM 
code) occurring at least 7 days— but no more than 
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59 days— after the date of initiation of chemotherapy. If a 
second chemotherapy cycle did not commence prior to 
day 60, both the first cycle of chemotherapy and the 
course of chemotherapy were considered to have been 
completed 30 days following the beginning of the cycle. 
The second and all subsequent chemotherapy cycles were 
similarly defined, up to a maximum of nine cycles in 
total. 

Study population 

The study population included all patients in the source 
population who were admitted to an acute care, short 
stay hospital in the GHS during the chemotherapy 
course— based on healthcare claims data— and who had 
an absolute neutrophil count (ANC) within 1 day of 
hospital admission (i.e., the day before, the day of, or the 
day after admission) -based on EMR data. Because 
chemotherapy-induced FN almost never occurs earlier 
than the fifth day of a chemotherapy cycle, and because 
some of the aforementioned retrospective studies did 
not consider hospitalizations during this period, only 
hospitalizations occurring on or after the fifth day of a 
chemotherapy cycle were included [11-13]. 

Febrile neutropenia 

The gold standard for identification of FN hospitalization 
(i.e., presumptive, for purposes of this retrospective evalu- 
ation) was evidence in EMR data of ANC <1.0 x 10 9 /L 
and either body temperature >38.3°C (101°F) or admi- 
nistration of antibiotic or antiviral therapy following 
ANC assessment, all having occurred within 1 day of 
hospitalization (i.e., the day before, day of, or day after 
hospitalization); this definition was considered the pre- 
sumptive "gold standard" for purposes of analyses. Receipt 
of antibiotics was included in the definition because some 
patients (12%) had no recorded body temperature data 
during the period within one day of hospital admission, 
while others may have had fever prior to admission that 
had resolved with antipyretic therapy. We chose our gold 
standard definitions of neutropenia and fever to be as con- 
sistent as possible with those employed by the National 
Comprehensive Cancer Network (NCCN) and Infectious 
Diseases Society of America (IDSA), given the limitations 
of our database. The NCCN/IDSA defines neutropenia as 
ANC <0.5 x 10 9 /L or ANC <1.0 x 10 9 /L with a predicted 
decline to <0.5 x 10 9 /L over the next 48 hours. Because 
the study database does not include information on antici- 
pated decline in ANC over time, and many patients— we 
believe— with ANC 0.5-1.0 would be expected to have 
ANC <0.5, we defined neutropenia as ANC <1.0 x 10 9 /L 
[17,18]. (We note that less authoritative guidelines have 
employed the higher threshold in their definition [19,20]). 
The NCCN/IDSA defines fever as a single temperature 
>38.3°C orally or >38.0°C over 1 hour. Because the study 



database does not provide information on duration of 
temperature readings, and because— we believe— the large 
majority of recorded readings were obtained orally, 
we defined fever as >38.3°C In addition, because body 
temperature readings were not consistently recorded for all 
patients, our definition of fever was expanded to include 
administration of antibiotic or antiviral therapy, which was 
assumed to be a proxy for presence of infection and fever. 

Claims-based operational definitions of FN hospitalization 
were based on ICD-9-CM inpatient diagnosis codes used in 
the aforementioned retrospective studies, and included: 

(1) neutropenia (ICD-9 288.0) in the primary position; 

(2) neutropenia in any position; (3) neutropenia plus fever 
(780.6) in any position; (4) neutropenia or fever in any 
position; and (5) neutropenia, fever, or infection in any pos- 
ition. (Because the last definition has been used in analyses 
to identify all neutropenic complications [including FN] 
resulting in hospitalization, its sensitivity was expected to be 
greater than the others, although at some loss of specificity 
and positive predictive value [PPV].) All of the aforemen- 
tioned claims-based definitions were set forth on a priori 
basis, and thus were "blinded" to evaluation of the gold 
standard; however, operational definitions considering 
the ICD-9-CM diagnosis code for pancytopenia (284.1) 
(i.e., either neutropenia or pancytopenia) were evaluated on 
a "post-hoc basis. 

Each hospitalization was classified as either an FN 
hospitalization (FN positive) or a non-FN hospitalization 
(FN negative) based on each of the alternative claims-based 
definitions and gold standard, respectively. Thus, for each 
comparison of a claims-based definition versus the gold 
standard, four mutually-exclusive and mutually-exhaustive 
results were possible: (1) true-positive (FN positive by both 
definitions); (2) false-negative (FN negative by claims-based 
definition, FN positive by gold standard); (3) true-negative 
(FN negative by both definitions); and (4) false-positive (FN 
positive by claims-based definition, FN negative by gold 
standard) (Table 1). 

Measures and analyses 

The accuracy of claims-based definitions for FN, relative 
to the gold standard, was evaluated based on their test 
characteristics and performance characteristics, principally 
PPV and sensitivity; specificity, negative predictive value, 
accuracy, and positive and negative likelihood ratios 
also were evaluated. Hospitalizations misclassified by 
the claims-based definitions (i.e., false-positives and false- 
negatives) were further investigated and their diagnosis 
codes were reported. Confidence intervals for estimates of 
test and performance characteristics were calculated using 
techniques of nonparametric bootstrapping (percentile 
method) from the study population (1,000 replicates with 
replacement). Analyses were conducted using SAS® Soft- 
ware, Release 9.2 (SAS Institute Inc., Cary, NC). 



Table 1 Definitions of test and performance characteristics 



Gold standard 





Yes No 




Yes 


X Y 




No 


Z K 




Test/Performance Characteristic 


Definition 


Explanation 


Patient Classification, % 






True-Positive 


X / (No. of Patients) x 1 00% 




False-Positive 


Y / (No. of Patients) x 1 00% 




True-Negative 


K / (No. of Patients) x 1 00% 




False-Negative 


Z / (No. of Patients) x 1 00% 




Sensitivity, % 


X/(X+Z) x 100% 


% of FN hospitalizations correctly identified by claims-based definition 


Specificity, % 


K / (Y+K) x 1 00% 


% of non-FN hospitalizations correctly identified by claims-based definition 


Positive Predictive Value (PPV), % 


X / (X+Y) x 1 00% 


% of hosp. identified as FN hosp. by claims-based definition that were true-positives 


Negative Predictive Value (NPV), % 


K/ (Z+K) x 100% 


% of hosp. identified as non-FN hosp. by claims-based definition that were true-negatives 


Accuracy, % 


(X+K) / (No. of Patients) x 100% 


% of all hospitalizations that were true-positives and true-negatives 


Likelihood ratio positive (LR+) 


[X/(X+Z)] / [1-K/(Y+K)] 


Odds of hosp. being an FN hosp. when claims-based algorithm is positive 


Likelihood ratio negative (LR-) 


[1-X/(X+Z)]/[K/(Y+K)] 


Odds of hosp. being a non-FN hosp. when claims-based algorithm is negative 



FN: febrile neutropenia. 
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Results 

A total of 3,193 patients received one or more courses of 
myelosuppressive chemotherapy during the period of inte- 
rest, were continuously eligible for comprehensive medical 
and drug benefits during the 60 days prior to initiation of 
chemotherapy, and had evidence of cancer (Table 2). 
Among these patients, 772 (24%) had a hospitalization 
during a course of chemotherapy on or after the fifth day 
of a new cycle, and of these patients, 357 (46%) were hospi- 
talized within GHS and had available data on ANC within 
1 day of hospitalization, and thus were included in the 
study population. No patient in the study population had 
more than one qualifying hospital admission during the 
study period. 

Mean age of the study population was 64 (SD: 13) years 
and 56% of patients were male (Table 3). Mean ANC at 
the time of hospital admission was, on an overall basis, 
4.4 x 10 9 /L. Common types of cancer included breast 
(10%), lung (19%), colon/rectum (11%), pancreas (8%), 
and non-Hodgkin's lymphoma (11%). Thirty-five percent 
of subjects had metastatic disease, and 68% received 
chemotherapy with >2 myelosuppressive agents. Hospita- 
lizations occurred most often in cycle 1 (28%) and cycle 2 
(17%). Among the 357 patients in the study population, 82 
(23%) met the gold standard for hospitalized FN; 28 (34%) 
of the 82 cases had low ANC, fever, and receipt of anti- 
microbial therapy, 30 (37%) had low ANC and receipt of 
antimicrobial therapy only (5 of the 30 were missing 
temperature data), and 24 (29%) had low ANC and fever 
only (all within 1 day of hospitalization). 

PPV was higher than 80% for claims-based definitions in- 
cluding the diagnosis codes for neutropenia plus fever in 
any position (PPV, 100%; sensitivity, 34% [95% CI: 24-45]) 
and neutropenia in the primary position (PPV, 87% 
[78-95]; sensitivity, 57% [46-68]) (Table 4). PPV was 77% 
(68-87) for the claims-based definition including the 
diagnosis code for neutropenia in any position; sensitivity 
was 67% (56-77). Sensitivity was highest 87% (79-94) when 
identifying hospitalized FN based on a diagnosis code for 
neutropenia, fever, or infection; PPV was 35% (28-42). The 
addition of the code for pancytopenia to the alternative 

Table 2 Selection of source and study populations 

Inclusion/exclusion criteria 



claims-based definitions had minimal impact (i.e., it chan- 
ged PPV and sensitivity by <3%). 

For the claims-based definition identifying FN hospi- 
talization based on a diagnosis of neutropenia in any 
position, eight of the 27 false-negatives patients had a 
diagnosis of septic shock/severe sepsis/septicemia, two 
had a diagnosis of pancytopenia, five had diagnoses of 
other infections, and 12 had neither a diagnosis of infec- 
tion nor fever. Among the 16 false-positives, six patients 
had neutropenia (ANC< 1,000), but none received antimi- 
crobials and three had no temperature data recorded in 
the relevant time period. In addition, two patients, of 
whom one was febrile, had an ANC > 1,000 but < 1,500, 
and one had fever and received antimicrobial therapy with 
an ANC of 1,617. 

Discussion 

Because they are easily accessible and provide readily- 
available information on a large number of patients 
receiving care in real-world clinical practice, private and 
public healthcare claims have been used extensively to 
date in health-economic and outcomes research. Such 
databases, however, are not without limitations, since 
they are not designed to capture certain components of 
care (e.g., laboratory results) and thus may lack data on 
potentially clinical parameters, which oftentimes are 
important for case-ascertainment. Inaccuracies in case- 
ascertainment may confer unknown biases in the estima- 
tion of study results. 

In this study, we evaluated the accuracy of operational 
definitions previously used to identify chemotherapy- 
induced hospitalization FN in healthcare claims data- 
bases. We found that PPV exceeded 80%-the assumed 
minimum acceptable threshold— for the definitions 
including the diagnosis code for neutropenia in the prin- 
cipal position (87%) and diagnosis codes for neutropenia 
plus fever (100%), and that PPV was close to the accept- 
able threshold for the definition including a diagnosis 
code for neutropenia in any position (77%). Among these 
definitions, sensitivity was highest for the last one (67%), 
although this figure is somewhat lower than that (81%) 

n (%) 



Source Population 

GHP members aged >18 years who received myelosuppressive chemotherapy from 01/01/04 - 4/30/10 
plus continuous health benefits during 60-day pretreatment period 

plus >2 claims with primary cancer diagnosis beginning 60 days prior to chemotherapy initiation 
Study Population 

plus admission to acute care, short stay hospital during chemotherapy course 
plus admitted to hospital within GHS 
plus ANC data available within 1 day of hospital admission 
GHP: Geisinger Health Plan; GHS: Geisinger Health System; ANC: absolute neutrophil count. 



3823 (100.0) 
3660 (95.7) 
3193 (83.5) 

772 (20.2) 
373 (9.8) 
357 (9.3) 



Table 3 Patient, cancer, and chemotherapy characteristics 



Patient 
Age (years), mean±SD 
Male, % 

Body Mass Index (kg/m 2 ), mean±SD 
N 

Absolute Neutrophil Count 
(cells/L), mean±SD 

N 

Cancer 
Type % 

Bone, Connective Tissue, Skin, Breast 
Female Breast 
Other 

Respiratory and Intrathoracic Organs 
Trachea, Bronchus, Lung 
Other 

Digestive Organs and Peritoneum 
Colon/Rectum 
Pancreas 
Esophagus 
Other 

Non-Hodgkin's Lymphoma 
Genitourinary Organs 

Bladder 

Ovarian 

Prostate 

Uterus 

Other 



All 

patients 
(n=357) 



64.2+1 3.0 

56.0 
27.3±5.9 

332 
4.4±5.6 

357 



9.8 
2.8 

19.3 

I. 4 

10.9 
7.8 
2.5 
3.1 

II. 2 

4.2 
3.6 
2.8 
1.7 
2.8 



Gold 
standard 
(n=82) 



62.1 ±13.8 

51.2 
27.8±4.5 

// 
0.2±0.3 



82 



19.5 
1.2 

17.1 
2.4 

9.8 
1.2 
2.4 
0.0 
20.7 

3.7 
1.2 
0.0 
0.0 
1.2 



FN Patients: 
neutropenia, 
primary position 
(n=54) 



63.3+15.1 

42.6 
27.8±4.7 

51 
0.3±0.9 



54 



18.5 
1.9 

9.3 
5.6 

5.6 
1.9 
0.0 
0.0 
25.9 

3.7 
5.6 
0.0 
0.0 
1.9 



FN Patients: 
neutropenia, 
any position 
(n=71) 



62.6±14.8 

40.9 
28.1 ±4.7 

67 
0.9±3.3 



71 



18.3 
2.8 

9.9 
4.2 

5.6 
2.8 
0.0 
0.0 
23.9 

4.2 
4.2 
0.0 
1.4 
1.4 



FN Patients: 
Neutropenia + 
Fever,Any Position 
(n=28) 



61.3±16.3 

39.3 
28.1+5.7 

28 
0.2±0.2 



28 



28.6 
3.6 

14.3 
3.6 

7.1 
0.0 
0.0 
0.0 
28.6 

0.0 
0.0 
0.0 
0.0 
0.0 



FN Patients: 
neutropenia or 
fever, any position 
(n=78) 



63.0+14.5 

44.9 
28.2+5.1 

74 
1.1+3.5 



'8 



17.9 
2.6 



10.3 



5.1 
5.1 
0.0 
0.0 
244 

3.8 
3.8 
0.0 
1.3 
1.3 



FN Patients: 
neutropenia, fever, or 
infection, any position 
(n=201) 



65.5+12.3 

56.2 
27.3+5.2 

192 
4.4+6.3 



201 



11.9 

2.5 

20.9 
2.0 

9.5 
7.0 
1.0 
2.0 
14.9 

5.0 
3.0 
3.0 
0.5 
2.0 



Table 3 Patient, cancer, and chemotherapy characteristics (Continued) 



Lip, Oral Cavity, and Pharnyx 2.0 1.2 0.0 

Blood Cancers 6.7 13.4 13.0 

Hodgkin's Lymphoma 1.7 1.2 1.9 

Other 5.6 3.7 5.6 

Presence of Metastases, % 35.0 24.4 22.2 
Chemotherapy 

No. of Myelosuppressive Drugs in Cycle 1, % 

1 32.2 20.6 26.1 

2 47.2 50.7 45.7 
>3 20.6 28.8 28.3 

Year of Initiation, % 

2004-2005 41.7 37.8 42.6 

2006-2007 28.3 28.1 22.2 

2008-2010 30.0 34.2 35.2 



0.0 
14.1 
1.4 
5.6 
21.1 



27.4 
46.8 
25.8 

42.3 
23.9 
33.8 



0.0 
14.3 
0.0 
0.0 
17.9 



38.1 
38.1 
23.8 

14.3 
32.1 
53.6 



0.0 
14.1 
1.3 
5.1 
20.5 



29.4 
44.1 
26.5 

39.7 
23.1 
37.2 



1.5 
8.0 
0.5 
5.0 
26.4 



29.2 
49.7 
21.1 

37.3 
31.4 
31.3 



b- a. 

i 1 

as ^ 

a. n 

n -p 

3 a. 



n Ln 
o 3 
3 2. 



Si! S 

o> — ' 



Table 4 Test and performance characteristics of claims-based definitions for febrile neutropenia 



Gold standard: 

FN: No FN: FN: No FN: FN: No FN: FN: No FN: FN: No FN: 

Claims-Based FN: 47 7 FN: 55 16 FN: 28 0 FN: 55 23 FN: 71 130 

Definitions: No FN: 35 268 No FN: 27 259 No FN: 54 275 No FN: 27 252 No FN: 11 145 



Test/Performance Characteristic* 
Patient Classification, % 

True-Positive 

False-Positive 

True-Negative 

False-Negative 
Sensitivity, % 
Specificity, % 

Positive Predictive Value (PPV), °A 
Negative Predictive Value (NPV), 
Accuracy, % 

Likelihood ratio positive (LR+) 
Likelihood ratio negative (LR-) 



Neutropenia, 
Primary Position 



13% (10-17) 

2% (0-3) 
75% (70-79) 
10% (7-13) 
57% (46-68) 
97% (96-99) 
87% (78-95) 
88% (85-92) 
88% (85-91) 
22.5 (12.3-68.5) 
0.4 (0.3-0.6) 



Neutropenia, 
Any Position 



15% (11-19) 

4% (2-7) 
73% (68-77) 
8% (5-10) 
67% (56-77) 
94% (91-97) 
77% (68-87) 
91% (87-94) 
88% (85-91) 
1 1 .5 (7.6-20.4) 
0.3 (0.2-0.5) 



Neutropenia + 
Fever.Any Position 



8% (5-11) 

0% ( ) 

77% (73-81) 
15% (11-18) 
34% (24-45) 

100% ( ) 

100% ( ) 

84% (80-87) 
85% (82-89) 

0.7 (0.6-0.8) 



Neutropenia or 
Fever, Any Position 



15% (11-19) 

6% (4-9) 
71% (66-75) 
8% (5-10) 
67% (56-77) 
92% (88-95) 
71% (61-80) 
90% (87-94) 
86% (82-90) 
8.0 (5.6-13.4) 
0.4 (0.3-0.5) 



Neutropenia, Fever, or 
Infection, Any Position 



20% (16-24) 
36% (31-42) 
41 % (36-45) 

3% (1-5) 
87% (79-94) 
53% (47-59) 
35% (29-42) 
93% (89-97) 
61 % (55-66) 
1.8 (1.6-2.1) 
0.3 (0.1-0.4) 



FN: febrile neutropenia. 
•Value (95% CI). 
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reported for a similar definition in the aforementioned 
evaluation of older adults with non-Hodgkin's lymphoma 
[15]. While sensitivity was improved in our study by in- 
cluding codes for fever and infection (87%), PPV (35%) was 
lower. We note, however, that this claims-based definition 
has most often been employed in comparative effectiveness 
studies of CSF agents, in which the focus was not on FN 
per se but all neutropenic complications (including severe 
neutropenia, afebrile infections in neutropenic patients, as 
well as FN) resulting in hospitalization that may be 
prevented with prophylaxis. When modifying the gold 
standard (i.e., ANC was < 1.5 x 10 9 /L) to capture additional 
neutropenic complications-irrespective of whether the 
patient was febrile or received antimicrobials-PPV (45% 
[38-53]) improved, but sensitivity was lower (81% 
[74-88]). 

Because no one definition was found to be associated 
with both high PPV and high sensitivity, the selection of 
a specific definition for identifying hospitalized FN 
should be guided by the potential impact of misclassifi- 
cation on study outcomes. For example, in studies evalu- 
ating the comparative effectiveness of CSFs in clinical 
practice, a premium should be placed on a high PPV and 
limiting the inclusion of false-positive cases. Since CSFs 
can only reduce the risk of infections associated with 
true neutropenia, a low PPV would result in a dilution 
of the estimated effectiveness of CSF prophylaxis and 
estimated relative differences in effectiveness among 
agents being studied. A high PPV alone may not be suffi- 
cient, however, since if sensitivity is low, analyses may not 
be adequately powered to evaluate study objectives. For 
other studies, however, such as those in which the outcome 
is the cost of FN hospitalization, some misclassification 
may be acceptable to the extent that there are offsetting 
effects (e.g., if sensitivity and PPV are comparable; and 
either costs of false-positives, false-negatives, and true- 
positives are similar or costs of false-negatives and false- 
positives differ in the opposite direction by the same 
degree). The same also may be true for studies characteriz- 
ing the clinical risk (i.e., incidence) of FN, for which— given 
the limitations of the alternative definitions— the goal 
should be to select one that is adequately sensitive and/or 
yields a balance between the number of patients who are 
misclassified (i.e., as false-positives and false-negatives). It 
therefore is essential to consider the importance of the 
individual test and performance characteristics of a mea- 
sure, relative to the objectives of the study, before selecting 
one definition for use, and we recommend considering sev- 
eral alternative definitions. 

Our study has a number of limitations. First, we ex- 
cluded from the study population all patients who were 
not hospitalized in a GHS facility (because ANC data were 
not available for these patients) and those hospitalized in a 
GHS facility who did not have ANC data proximal to 



hospital admission. Only 35% of patients in the source 
population who were hospitalized (including patients trea- 
ted inside and outside GHS) had ANC data during the 
study time period and thus could be included in the study 
population. For these reasons, the number of patients in 
the study population was limited to those treated at a 
GHS facility and who had ANC data. To the extent that 
excluded patients actually had FN and differed from the 
study population in important respects, our results may 
not be reflective of the larger population. In addition, be- 
cause patients who had ANC data proximal to hospital ad- 
mission may be selectively different from those who did 
not (because of, for example, a history of FN), our study 
population may be "enriched" and subject to verification 
bias. Second, even for some patients included in the study 
population, critical data-especially body temperature-was 
missing and thus we used a gold standard based on ANC 
and body temperature or administration of antibiotic or 
antiviral therapy. Because we used an "imperfect" gold 
standard, however, and thus patients' true status in terms 
of FN was unknown, the presence of errors in the classifi- 
cation of patients by the gold standard could result in 
biased estimates of PPV, sensitivity, and other markers. 
Third, since the data for the study come from a single 
health plan and coding practices may vary across plans 
and geographic regions of the country, the generalizability 
of our findings is uncertain. Fourth, the accuracy of 
claims-based definitions in identifying FN may vary across 
subgroups of patients (e.g., those defined on the basis of 
type of cancer or regimen), and thus caution should be 
employed when generalizing study results to specific 
populations; such subgroup analyses, however, were not 
possible given the relatively small size of the study popula- 
tion. Fifth, while acknowledging that culture data are less 
than perfectly sensitive in identifying infection, the lack of 
such information in the data extract precluded any evalu- 
ation of the presence of infection in true FN patients 
based on positive cultures. Finally, because chemotherapy- 
induced FN almost never occurs earlier than the fifth day 
of a chemotherapy cycle, and because some of the afore- 
mentioned prior studies did not consider hospitalizations 
during this period, only hospitalizations occurring on or 
after the fifth day of a chemotherapy cycle were included. 
Results were not sensitive to this methodological feature 
as <5% of all hospitalizations occurred during the first five 
days of a chemotherapy cycle. 

Conclusion 

In summary, the results of our study suggest that patients 
hospitalized for chemotherapy-induced FN can be identified 
in healthcare claims databases-with an acceptable level of 
mis-classification-using the diagnosis code for neutropenia 
or diagnosis codes for neutropenia plus fever. Research 
investigating the test and performance characteristics of 
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operational definitions of FN using alternative databases 
(especially those in which ANC, body temperature, and cul- 
ture data are available) is needed to verify the findings of 
this study, as is research in tumor-specific subgroups. 
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